Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨Autoscaling: add buffer metrics #6260

Merged
merged 20 commits into from
Aug 29, 2024
Merged

✨Autoscaling: add buffer metrics #6260

merged 20 commits into from
Aug 29, 2024

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Aug 28, 2024 •
edited
Loading

What do these changes do?

This adds metrics for the buffer pooling:

  • in essence in the next days, every type of EC2 allowed to auto-scale will/can have a different number of buffer machines available,
  • for monitoring this PR brings a few prometheus metrics to check the status of the buffering pools
    For example:
    • we decide to have a pool of 4 buffers of type g4dn.xlarge,
    • autoscaling creates the said pool, these undergo different states (e.g. launched, ready-to-pre-pull-images, pulling images, stopping, ready to be used)
  • for monitoring performance prometheus Histogram were also added in order to measure the times for:
    • launching a machine until it is ready to pre-pull
    • pre-pulling images

--> Next step will be adding these in the grafana dashboard

bonus:

  • fixes Gauges not set back to 0 (they were cleared, which prometheus does not handle nicely)
  • refactoring

Related issue/s

How to test

Dev-ops checklist

@sanderegg sanderegg added the a:autoscaling autoscaling service in simcore's stack label Aug 28, 2024
@sanderegg sanderegg self-assigned this Aug 28, 2024
@codecov Codecov
Copy link

codecov bot commented Aug 28, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 98.47716% with 3 lines in your changes missing coverage. Please review.

Project coverage is 88.7%. Comparing base (cafbf96) to head (4292377).
Report is 483 commits behind head on master.

Files with missing lines Patch % Lines
...rvice_autoscaling/modules/instrumentation/_core.py 91.6% 2 Missing ⚠️
...e_service_autoscaling/modules/auto_scaling_core.py 75.0% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #6260      +/-   ##
=========================================
+ Coverage    84.5%   88.7%    +4.1%     
=========================================
  Files          10    1045    +1035     
  Lines         214   46592   +46378     
  Branches       25     406     +381     
=========================================
+ Hits          181   41352   +41171     
- Misses         23    5170    +5147     
- Partials       10      70      +60
Flag Coverage Δ
integrationtests 64.8% <ø> (?)
unittests 86.0% <98.4%> (+1.5%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...ackages/aws-library/src/aws_library/ssm/_client.py 97.7% <100.0%> (ø)
...oscaling/src/simcore_service_autoscaling/models.py 100.0% <100.0%> (ø)
...e_autoscaling/modules/buffer_machines_pool_core.py 88.3% <100.0%> (ø)
...e_autoscaling/modules/buffer_machines_pool_task.py 100.0% <ø> (ø)
...ce_autoscaling/modules/instrumentation/__init__.py 100.0% <100.0%> (ø)
..._autoscaling/modules/instrumentation/_constants.py 100.0% <100.0%> (ø)
...autoscaling/modules/instrumentation/_ec2_client.py 100.0% <100.0%> (ø)
...ice_autoscaling/modules/instrumentation/_models.py 100.0% <100.0%> (ø)
...vice_autoscaling/modules/instrumentation/_utils.py 100.0% <100.0%> (ø)
.../src/simcore_service_autoscaling/utils/rabbitmq.py 100.0% <ø> (ø)
... and 2 more

... and 1029 files with indirect coverage changes

@sanderegg sanderegg force-pushed the autoscaling/add-buffer-metrics2 branch 2 times, most recently from f3fceff to f39a17b Compare August 29, 2024 09:15
@sanderegg sanderegg marked this pull request as ready for review August 29, 2024 11:44
@sanderegg sanderegg added this to the Eisbock milestone Aug 29, 2024
odeimaiz
odeimaiz reviewed Aug 29, 2024
View reviewed changes
Copy link
Member

@odeimaiz odeimaiz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

mrnicegyu11
mrnicegyu11 approved these changes Aug 29, 2024
View reviewed changes
Copy link
Member

@mrnicegyu11 mrnicegyu11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left some minor comment but looks very nice and fancy thx

packages/aws-library/src/aws_library/ssm/_client.py Show resolved Hide resolved
services/autoscaling/src/simcore_service_autoscaling/modules/buffer_machines_pool_task.py Show resolved Hide resolved
services/autoscaling/src/simcore_service_autoscaling/modules/buffer_machines_pool_core.py Show resolved Hide resolved
services/autoscaling/src/simcore_service_autoscaling/modules/instrumentation/_utils.py Show resolved Hide resolved
services/autoscaling/src/simcore_service_autoscaling/modules/instrumentation/_ec2_client.py Show resolved Hide resolved
@sanderegg sanderegg force-pushed the autoscaling/add-buffer-metrics2 branch from 005b8e5 to 6c50edc Compare August 29, 2024 12:44
@sanderegg sanderegg force-pushed the autoscaling/add-buffer-metrics2 branch from 6c50edc to 4292377 Compare August 29, 2024 14:50
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
0 New issues
1 Accepted issue

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@sanderegg sanderegg merged commit 7336ce5 into ITISFoundation:master Aug 29, 2024
56 checks passed
@sanderegg sanderegg deleted the autoscaling/add-buffer-metrics2 branch August 29, 2024 15:53
This was referenced Sep 4, 2024
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@odeimaiz odeimaiz odeimaiz left review comments

@mrnicegyu11 mrnicegyu11 mrnicegyu11 approved these changes

@YuryHrytsuk YuryHrytsuk YuryHrytsuk approved these changes

@pcrespov pcrespov Awaiting requested review from pcrespov

@giancarloromeo giancarloromeo Awaiting requested review from giancarloromeo

@mguidon mguidon Awaiting requested review from mguidon

@matusdrobuliak66 matusdrobuliak66 Awaiting requested review from matusdrobuliak66

Assignees

@sanderegg sanderegg

Labels
a:autoscaling autoscaling service in simcore's stack
Projects
None yet
Milestone
Eisbock
4 participants
@sanderegg @mrnicegyu11 @odeimaiz @YuryHrytsuk